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ABSTBACT 

It is argued that the role of the evaluator is fluid 
and subject to unpredictable change over time. Tvo icey principles 
Ifrom systems theory support the above: (1) the definition Df an 
organization as an open system, and (2) the self-regulatory or 
cybernetic nature of systems. Actual experiences are dravn upon to 
describe that interrupted the evaluation process ov changed the 
nature of the evaluation. The "By the way..." phenomenon referred to 
is described as an interaction between the organizition anl the 
evaluator that results in a modification, revision, aeletion or 
addition to the previously prescribed evaluator role and its 
attendant activities. The basic premise, that changes themselves ace 
not able to be anticipated, is explored: and it is concluded that the 
task becomes one of skill development (Specifically, renegDtiatlDnl 
so that the processes are there to handle the role changes as tbay 
occur. (Author/GK) 
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The meeting had gone well. The new evaluator had presented a 
draft of the pretest-posttest control -group design to the staff, and the 
program coordinator had been well pleased. The evaluator was ready to 
begin implementation of the design. To conclude the meeting, the eval- 
uator asked if there were any points that needed further discussion. 
There was only one. 

The coordinator turned to the evaluator and casually said, *'0h, 

by the way It wasn't in the schedule, but we've already started the 

intervention. Do you think it will affect the evaluation?" 



This paper is the product of the shared experiences of faculty 
and students in the Practicum in Program Evaluation of the Department of 
Educational Psychology, The University of Texas at Austin. Students in 
that course undertake evaluation responsibilities at human service agencies 
in the community and apply such skills as negotiating an evaluation agree- 
ment, selecting an appropriate design, collecting and analyzing data, and 
working with managers to facilitate use of the findings. In this context 
we noted repeated events that interrupted the evaluation process or 
substantially changed the nature of the evaluation itself. At first these 
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were viewed as idiosyncratic instances, and we approached them from a 
trouble-shooting perspective with the goal of solving the immediate prob- 
lems. Over the course of the semester, however, the frequency of their 
occurrence led us to treat these events as "facts of life" for the evaluatcr 
and we moved into the second semester by systematically documenting these 
events that occurred "By the way..." and structuring hypotheses so that 
our experiences could offer confirming or disconfirming evidence. These 
experiences, plus relevant theory from the literature that helps to under- 
stand the phenomenon, form the basis of this paper. 

BASIC DEFINITIONS 

The "By the way..." phenomenon refers to an interaction between 
the organization and the evaluator that results in a modification, revi- 
sion, deletion or addition to the previously prescribed evaluator role and 
its attendant activities. The "By the way..." event is typically unanti- 
cipated and unexpected by the evaluator (and perhaps by the administrator 
or manager as well). It is equally likely to occur when the evaluation 
is internal or external to the organization and it may be perceived by the 
evaluatoras a positive or negative experience. "By the way...s" have been 
observed at all stages of the evaluation process: (1) initiation and . 
contract negotiation, (2) conceptualization of formal design (3) design 
implementation/data collection, (4) data transformation, analysis, and 
interpretation, and (5) reporting, dissemination and utilization of findings. 

While "By the way..." events themselves vary widely in content, 
they have a common outcome: a redefinition of the role that the evaluator 
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is expected to perform in the organization. To understand the "By the 
way,,.) phenomenon better, it is first useful to examine past attempts to 
define the role of program evaluators. 

EVALUATOR ROLES 

A review of recent evaluation literature yields three distinct 
conceptual approaches to the definition of the evaluator role: (1) defi- 
nition by integration with management decision-making, (2) distinguishing 
generalist roles from specific evaluator roles, and (3) definition by 
level of program development (Kirkhart, 1980). 

First, the definition of evaluator role by integration with mana- 
gement decision-making is best illustrated by the work of Attkisson and 
colleagues in community mental health evaluation (Attkisson et al., 1978). 
A four-stage model is proposed, representing evolving levels of evaluator 
involvement with management. The least integration with management occurs 
at level one, Statistical/Clerical, where the evaluator functions as a 
statistical clerk. At the second level Clinical/Experimental, the eval- 
uator has higher technical competence but conducts only isolated experi- 
ments in the clinical setting. At the third level, Technical/Evaluative, 
the evaluator collects data requested by managers, and at the fourth and 
highest level of integration with management the evaluator him/herself is 
part of the decision-making process that dete^tnines what information is to 
be collected. While this model defines a variety of evaluator roles as 
legitimate, the roles representing greater integration of evaluation with 
decision-making are implicitly seen as more, desirable. Also noteworthy 
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for our purposes is the fact that the evolution of evaluator roles is hypo- 
thesized to follow an orderly progression upward across levels over time, 
but at any given point in time the evaluator may be identified with a 
fixed role. 

The second approach to defining evaluator role*^ addresses the 
question of whether such roles are generic or specific (McCul lough, 1975; 
Tornatzky, 1979). McCul lough (1975) argues in favor of the "general prac- 
titioner evaluator," i.e,, one who has the skills to accommodate a number 
of roles with ease. Like Attkisson, McCul lough points to a need to train 
several levels of evaluators to address the differing needs of organiza- 
tions, but he implies no evolution of roles within an organization. 
Tornatzky (1979) also omits a time dimension from his discussion of evalua- 
tor roles, proposing that evaluators simultaneously function as researchers, 
innovators, and politician/administrators. Moreover, Tornatzky discour- 
ages the fragmentation of evaluator roles, arguing that the general role 
of evaluator cannot be meaningfully separated from the more specific 
roles. 

Third, the evaluator role has been defined in terms of program 
development (Tripodi, Fellin, & Epstein, 1971, 1978; Johnson, 1970). 
These models propose that the role of an evaluator within a program is 
determined by that program's developmental stage. Thus within a given 
developmental stage, such as initiation, contact. Implementation (Tripodi, 
Fellin, & Epstein, 1971, 1978) or formulation, implementation, and post- 
project (Johnson, 1970) the evaluator role remains fixed; however the 
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evaluator's role would be expected to change in a predictable manner as 
the program itself evolves. 

In all cases, the frameworks proposed to date for addressing eval- 
uator roles assume that the role of an evaluator is either fixed at any 
given point in time or that it will vary predictably according to certain 
criteria which can be identified in advance. This, in turn, is based on 
an even larger assumption: the organization of which the evaluator is a 
member will remain fixed or will vary predictably over time. It is our 
contention that neither of these assumption is borne out in actual practice 
and that present models are inadequate to explain the functional ro'.e of 
evaluators. 

We would argue that evaluator roles are, in fact, fluid and not 
fixed. The static models of evaluator roles have forced clarity out of 
complexity, presenting a theoretical "neatness" that does not exist in 
practice. Moreover, we contend that it is impossible to anticipate all 
eventualities such that a clearcut initial role definition is possible. 
Rather, the definition of the evaluator's role is an interactive process, 
and redefinition and adjustment of his/her role are continuously occurring 
in response to unforseen events; hence, the "By the way..." phenomenon. 
These events are not seen as idiosyncratic aberrations in the organization 
but as normal occurrences in the operation of open systems. To under- 
stand the "By the way..." phenomenon and the role of the program evaluator 
then, one must turn to systems theory. 
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EVALUATION IN OPEN SYSTEMS 



The rnodel of an organization as a system provides a theoretical 
base whiwh supports our assumption of change and fluidity within organi- 
zations as a whole and within evaluation as a specific subsystem. Two 
key principles emerge from systems theory that support our argument that 
the role of evaluator is fluid and may change unpredictably over time. 
They are: (1) the definition of an organization as an open system, and 
(2) the self-regulatory or cybernetic nature of systems. 

The first principle defines the organization as an open systen,. 
Since human service delivery systems (the evaluation setting of concern 
here) maintain an ongoing exchange with their environments--taking in, 
processing, and sending out resources, information and material s--they 
are considered open systems. More specifically, open systems are defined 
by three assumptions (Baker, 1969). First, organizations are defined by 
their primary task; that is, the task which the organization was created 
to perform. In human sercice systems, the definition of primary task is 
frequently complicated by the existence of multiple goals, some of which 
cannot be accomplished without interaction with other systems. Second, 
as an open system, an organization accomplishes its primary task by admit- 
ting inputs from the environment, using its subsystems to convert them, 
and sending outputs back into the environment. Third, organizdtionb 
encounter boundary conditions which rapidly change the characteristics 
of the organization. Boundary conditions refer to the various dimen- 
sions of the interorganizational environment which provide inputs to or 
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accept outputs from a given organization and which affect the organiza- 
tion's structure and functioning (Baker, 1959). The concept of the organ- 
ization as one system interacting with other systems is an important 
point. It implies, for example, that the ability of Organization A to 
anticipate fluctuations in its operation can be determined only to the 
extent that the operation:^ of the other organizations in its environment 
can be accurately predicted. However, Organization A has a limited capa- 
city to predict the operations of the other organizations in its environ- 
ment. Thus, it cannot predict when or exactly how it will have to respond 
or adjust its functioning to adapt to inputs from these other systems. 
A systems view of this adaptation process takes us to the sacond prin- 
ciple: the cybernetic nature of systems. 

All system parts are connected by processes devoted to maintain- 
ing the integrity (growth, stability, survival) of the organization 
during periods of normal functioning as well as times of change. These 
mechanisms are homeostatic and self-regulotory in their actions. That 
is, they act to signal the system when its functioning exceeds or is c.bout 
to exceed the limits of its normal range of operation, and they also act 
to return the system to within this normal range. These mechanisms adjust 
and redefine the functions of the subsystems to maintain system inte- 
grity. 

As we noted previously, the organization cannot alv/ays predict 
the timing or nature of Its environmental inputs and therefore, cannot 
predict with certainty the course of its own functioning over time. The 
organization's chief means of adaptation to this uncertainty is to change 
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tUe function(s) of one or more of its subsystems. The changes in function 
modify, in whole or in part, the subsystem delegated to handle the input. 
The modification in function is represented at the level of the indivi- 
dual as a change ir^ one's role within the organization. 

In summary, because the organization is dependent on other systems 
in its environment, it can neither remain static nor c&n it perfectly 
predict changes in state. Inasmuch as the subsystems operate according to 
the same principles as the larger system, they also neither remain fixed 
nor vary predictably over time. These unpredictable subsystem changes are 
passed on to the level of the individual as role redefinition. 

While the specific events which impinge on and necessitate rede- 
finition of the evaluator role cannot always be anticipated based on the 
information available to the evaluator, systems theory does provide a 
perspective for identifying critical setting variables for the occurrence 
of "By the way..." events. We propose that the following eight dimen- 
sions affect the extent (frequency and degree) of role redefinition: 

1. Organizational congruence . The degree to which the organi- 
zation has effectively and efficiently integrated the struc- 
ture and functioning of its various components. 

2. Task clarity . Degree of clarity of the evaluation task 
itself. 

3. Administrative competence and functioning . The extent to 
which authority and responsibility are clearly delegated, 
feedback loops established, clear messages given, and organi- 
zational goals are consistently communicated. 

4. Procedures for adaptation . The existence and adequacy of 
procedures that an organization can call into action when 
faced with an unexpected input. 

5. Evaluator characteristics . The expertise of the evaluator 
plus his/her diplomacy and personality. 
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6 . Organizational commitment to the evaluation task/goals . 
The extent to which evaluation is an integral part of the 
organization, well integrated into its structure and func- 
tioning. 

7. Internal input into the organization . How well the infonrial 
and formal structure of the organization fulfills mutual and 
individual goals. 

8. External input into the organization . Changes in the larger 
environment which act to change the nature or functioning 

of the organization. 

To return to our opening vignette, the following case study illus- 
trates the interaction of these factors in setting the stage for a "By 
the way...". (No doubt the reader may know of situations of equal or 
greater difficulty or perhaps ones that led to positive redefinitions 
of role, ) 

The local public school system had decided to establish a 
one-year nutrition education program. The program was initiated 
and planned by the local Parent-Teacher Association and school 
officials before the evaluator was hired. During the contract 
negotiations, the evaluator and school officials defined the 
evaluator' s responsibilities which included designing pre- and 
post-tests to be used, collecting baseline data, and collecting 
ongoing evaluation data on the effects of the curriculum on 
students' eating habits at lunch. The contract was agreed on 
and the evaluator reported to work soon thereafter. 

His early days on the job were replete with "By the way,,.s". 
Due to poor lines of communication among the nutrition program 
sraff, tha school administration and the school faculty, the 
program was already being piloted in one of the schools in the 
district. A full five months of nutrition instruction had been 
given to the target classes without the nutrition program 
director's knowledge. The first "By the way..." request for the 
evaluator was to create a design that could take into account 
the previous "accidental" nutrition instruction. 

The second big surprise soon came when the evaluator dis- 
covered that the nutrition project director had already allowed 
the experimental lunchroom programs to start, thus precluding 
the gathering of "clean" baseline information on students' 
nutritional habits at lunch. 
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A third "By the way..." came into the picture from outside 
the system in the form of court-ordered desegregation. Con- 
siderable amt'iguity existed at that time in the nature of the 
desegregatior. plan, when it was to be implemented and which 
schools were to be affected. The stakes were high for all 
concerned, and the efforts of school personnel and administra- 
tion were marshalled to deal with potential problems. Attention 
and commitment to th^ nutrition project i>egan to wane. Further, 
since the schools were unclear as to how they were going to 
be affected by the desegregation plan, the nutrition project 
director had no way of knowing what effects it would have on 
the nutrition program. School district personnel could offer 
no guidance, as they were unsure as well. The e\/aluator then 
found ^nms.'^lf asked not to set up and implement an evaluation 
design -^or a program, but to develop a series of design options 
contingent upon which desegregation plan was adopted. 

A number of explicit "By the way..." examples emerge from this 
case study that serve to illustrate how the dimensions listed previously 
affect role redefinition. First, the teaching of the accidental nutrition 
I'^sson suggests a breakdown in the communication lines (congruence) of 
the organi2ation--a situation not uncovered until the evaluator had passed 
through the contract negotiation phase and gotten to hands-on work with 
the project. Regardless of the contract terms, .he problem was his to 
deal with. Next, one might question the competence of the director of 
the nutrition program with resoect to program evaluation for allowing 
the experimental lunchroom programs to begin before baseline data could 
be collected. This is especially pertinent since the responsibility for 
collecting s;ich data was clearly outlined in the evaluator's contract. 

Despite these internal problems, the major "By the way..." event 
Involved external input into the system, i.e., the court ordered deseg- 
regation. There was no way the school personnel, let alone the director 
of rt single project within a large urban school district, could have 
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anticipated this external input into the system. The school as a system 
had to respond to thir, input. Consequently, responsibilities of school 
personnel in other quarters (teachers, principals) shifted, and commitnient 
to the program shifted along with it. The district's lack of clarity 
about the desegregation plan also prevented the adoption of an effective 
adaptive procedure to deal with the potential changes in the organiza- 
tion of the schools with respect to the nutrition program. The evalua- 
tor found himself a reluctant participant in this entire process. There 
was little he could do other than to use his expertise to facilitate the 
best possible evaluation given the circumstances at hand. In fact, the 
characteristics of the evaluator helped here. The evaluator wac know- 
ledgeable about the school system and evaluation techniques and was sym- 
pathetic to the nutrition program director's plight. Her response to 
the lack of Direction from top levels of the organization was to draw as 
much from the evaluator's expertise as she could, indirectly giving him 
respor^sibility for tasks defined In her role as director of the program. 

In summary, the factors involved in this case of role redefini- 
tion included (1) poor organizational congruence, (2) lack of admini- 
strator competence with respect to evaluation, (3) ineffective or non- 
existent procedures for adaptation, (4) an unanticipated external input 
to the organization, (5) diminished commitment of personnel associated 
with the project, (6) evaluator characteristics, and (7) lack of task 
clarity once the unanticipated, external input occurred. 
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IMPLICATIONS 

While careful future research might lead to predictive models 
around these and other variables, such quantification is at this time 
prepiature. Rather, vie can talk in qualitative terms about a certain 
"coefficient of unexpectedness" which results from the presence or absence 
of the dimensions discussed above and their interaction. It is important 
to emphasize that rule definition and redefinition is not simply a matter 
of insightful contract negotiation or start-up planning. Rather, there 
are numerous countervailing forces operating during the entire course of 
an evaluation that may potentially act to redefine the nature of the 
task. For better or for worse, these forces impinge upon the evaluator's 
performance and definition of his/her role in that evaluation process. 
However, cognizance of these elements and the realization that "By the 
way..*s" can be expected to occur as a normal condition of the evaluation 
process will assist the evaluator in successfully responding in the most 
adaptive way to the redefinition of his/her evaluation activities. 

Moreover, the evaluator need not enter the process unprepared. 
The "By the way..." phenomenon has clear implications for training program 
evaluators. If our basic premise is correct--that the changes themselves 
are not able to be anticipated, then the task becomes one of skill develop- 
ment so that the processes are there to handle the re changes as they 
occur. Thus, role renegotiation skills must be a pe: c of the evaluator's 
armamentarium. 
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